Systems and methods for identifying objects and providing information related to identified objects

ABSTRACT

Systems and methods for identifying an object and presenting additional information about the identified object are provided. The techniques of the present invention can allow the user to specify modes to help with identifying objects. Furthermore, the additional information can be provided with different levels of detail depending on user selection. Apparatus for presenting a user with a log of the identified objects is also provided. The user can customize the log by, for example, creating a multi-media album.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Rosenblatt, U.S. Provisional PatentApplication No. 61/010,029 (Attorney Docket No. 104677-0146-001), filedJan. 3, 2008, entitled “Systems and Methods for Identifying Objects andProviding Information Related to Identified Objects,” the entirety ofwhich is incorporated herein by reference.

BACKGROUND OF THE INVENTION

This relates to portable electronic devices and more particularly tosystems and methods for identifying objects and for providinginformation related to the identified objects.

Portable electronic devices, such as wireless and cellular telephones,digital media players (e.g., music players and video players), andhybrid devices that combine telephone and media playing functionalityare known. These portable electronic devices can include, for example, acamera that may be capable of capturing an image of an object, but suchdevices cannot provide a user with other related information about theobject. For example, a user may be in an art museum looking at a pieceof art. Although the portable electronic device might be capable oftaking a picture of the art, the portable electronic device does nothave the capability to provide the user with detailed information aboutthe piece of art, such as the name of the painter or any other relatedinformation. In fact, the portable electronic device might not even havethe capability of identifying the object itself.

As another example, the user can use a portable electronic device tolisten to an audio tour that may provide recorded information aboutpieces of art. Although the user may be allowed to fast forward orrewind the audio tour, the information provided by the audio tour isfixed. Therefore, the user can not access information about other art inthe museum that has not been recorded as part of the audio tour.Moreover, even for the art that is included on the audio tour, the useris limited to the information that has already been recorded.

As still another example, the user can download podcasts about differentart exhibits on a portable electronic device. However, the informationprovided by the podcasts is still fixed because the user can notimmediately request for additional information located elsewhere.

Furthermore, while these portable electronic devices may be able todisplay pictures taken previously, such portable electronic devices maynot be able to provide a user with historical information about what theuser has already seen. For example, the user may have visited a zoo andmay have used the portable electronic device to take a picture of thefront entrance of the zoo. Although the user can show the picture to afriend days later, the user can not use the portable electronic deviceto find the zoo's location.

In view of the foregoing, it would be desirable to provide systems andmethods that provide a user with the capability to view an object'sidentity and obtain additional information about the identified object.It would also be desirable to provide a user with the capability toaccess a history of previously identified objects.

SUMMARY OF THE INVENTION

In accordance with the invention, a portable electronic device andmethod of operation are provided in which the device can attempt toidentify an object and obtain additional information about theidentified object. The portable electronic device can additionallyprovide a history of previously identified objects to the user.

In some embodiments, in order to detect an object, the portableelectronic device can automatically determine the current environmentand, based on the determined environment, select a detection technology(e.g., a RFID reader, a camera, and an IR image capture device). Forexample, if the portable electronic device detects active RFID tagsbroadcasting signals to the RFID reader, the portable electronic devicecan use the RFID reader to detect objects.

In some embodiments, the portable electronic device can allow the userto select a mode based on the types of objects that the user wants toidentify. Based on the selected mode, the portable electronic device canadjust parameters used for searching an identification database. Forexample, if the user selects to identify an object in a “MUSEUM” mode,the portable electronic device can search the identification databasefor objects that are commonly found in a museum. In some embodiments,the portable electronic device can determine the location of the user tohelp identify an object. For example, if the user is determined to be inLas Vegas and the portable electronic device is set to a “RESTAURANT”mode, the device can limit the search of the identification database torestaurants in Las Vegas.

In some embodiments, if the portable electronic device produces multiplecandidate identifications, the portable electronic device can select abest candidate identification based on certain criteria. In response tothe user correcting the selected identification, the portable electronicdevice can automatically modify the criteria based on the user input.

After an object has been identified, the portable electronic device canprovide additional information about the identified object. In someembodiments, the portable electronic device can search for theadditional information based on the previously-defined mode. In someembodiments, the portable electronic device can provide additionalinformation with incrementally increasing levels of detail (e.g., basic,detailed, and extended).

In addition, a log can be provided to the user, which includes a historyof previously identified objects. In some embodiments, the user cancustomize the log to display information in various ways. For example,the user may create a multi-media album based on objects that wereidentified in a particular mode. A self-guided museum tour can becreated from the multi-media album that provides the right amount ofinformation for the user.

Therefore, in accordance with the present invention, there are providedsystems and methods for providing a user with the identity of an objectand additional information about the identified object. These systemsand methods can include a portable electronic device that can makeautomatic adjustments to the identification process and adjust theadditional information provided based on user input.

There is also provided a portable electronic device apparatus forproviding a user with a log of previously identified objects. The usercan customize the log by, for example, creating a multi-media album.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features of the present invention, its nature andvarious advantages will be more apparent upon consideration of thefollowing detailed description, taken in conjunction with theaccompanying drawings in which:

FIG. 1 shows a simplified diagram of a portable electronic system whichmay be operated in accordance with one embodiment of the presentinvention;

FIG. 2 shows a simplified diagram of another portable electronic systemwhich may be used in accordance with one embodiment of the presentinvention;

FIG. 3 shows a simplified block diagram of a portable electronic deviceconstructed and used in accordance with one embodiment of the presentinvention;

FIG. 4 shows a schematic view of a communications system which can beoperated in accordance with one embodiment of the present invention;

FIG. 5 shows a schematic view of an illustrative display screen of ahome screen in accordance with one embodiment of the present invention;

FIG. 6 shows a schematic view of an illustrative display screen forselecting objects to identify in accordance with one embodiment of theinvention;

FIG. 7 shows a schematic view of an illustrative display screen forproviding configuration options in accordance with one embodiment of theinvention;

FIG. 8 shows a schematic view of an illustrative display screen of alisting of objects that have been detected and/or identified inaccordance with one embodiment of the invention;

FIG. 9 is a schematic view of an illustrative display screen for editingan object in accordance with one embodiment of the invention;

FIG. 10 shows a schematic view of an illustrative display screen forproviding basic information related to an identified object inaccordance with one embodiment of the invention;

FIG. 11 shows a schematic view of an illustrative display screenproviding a listing of detailed information related to an identifiedobject in accordance with one embodiment of the invention;

FIG. 12 shows a schematic view of an illustrative display screen ofdetailed information that includes media in accordance with oneembodiment of the invention;

FIG. 13 shows a schematic view of an illustrative display screen ofextended information related to an object in accordance with oneembodiment of the invention;

FIG. 14 shows a schematic view of an illustrative display screen forproviding a log of identified objects in accordance with one embodimentof the invention; and

FIGS. 15A-15D show a flowchart of an illustrative process for providinginformation related to an identified object in accordance with oneembodiment of the invention.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

FIG. 1 shows a simplified diagram of a portable electronic device whichmay be operated in accordance with one embodiment of the presentinvention. Persons skilled in the art will appreciate that one or moreelements of the described embodiments can be interchanged and/orcombined in the present invention. Portable electronic system 100 mayinclude portable electronic device 102. Portable electronic device 102is shown as including display component 104 and user input component106. However, other displays and user input components can also beutilized without departing from the spirit of the present invention.

Display component 104 is illustrated in FIG. 1 as a display screen thatmay be integrated into portable electronic device 102. Display component104 does not have to be integrated into portable electronic device 102and may also be external to portable electronic device 102. For example,display component 104 may be a computer monitor, television screen,and/or any other graphical user interface, textual user interface, orcombination thereof. Display component 104 may enable portableelectronic device 102 to playback the video portion of video content,and/or may serve as part of the user interface, displaying commandmenus, or serve any other suitable display functions.

User input component 106 is illustrated in FIG. 1 as a click wheel.Persons skilled in the art will appreciate that user input component 106may also be any other type of user input component or device, such as,for example, a mouse, keyboard, trackball, slider bar, one or morebuttons, portable electronic device pad, dial, or any combinationthereof. User input component 106 may also include a multi-touch screensuch as that described in Westerman et al., U.S. Pat. No. 6,323,846,issued Nov. 27, 2001, entitled “Method and Apparatus for IntegratingManual Input,” which is incorporated by reference herein in itsentirety. User input component 106 may emulate a rotary phone or amulti-button portable electronic device pad, which may be implemented ona touch screen or the combination of a click wheel or other user inputdevice and a screen. A more detailed discussion of such a rotary phoneinterface may be found, for example, in McKillop et al., U.S. patentapplication Ser. No. 11/591,752, filed Nov. 1, 2006, entitled “Touch Padwith Symbols based on Mode,” which is incorporated by reference hereinin its entirety.

FIG. 2 shows a simplified diagram of another portable electronic systemwhich may be used in accordance with one embodiment of the presentinvention. Portable electronic system 200 may include portableelectronic device 202, which may be, for example, a portable mediaplayer, cellular telephone (such as Apple's iPhone), personal organizeror any other portable electronic device. Portable electronic device 202can incorporate user interface component 204. User interface component204, as shown in FIG. 2, is a multi-touch screen that may function asboth an integrated display screen and as a user input device. Portableelectronic device 202 may also include button 206, which may be used inconjunction with user interface component 204. Persons skilled in theart will appreciate that additional buttons and/or other user interfacedevices may be used without departing from the spirit of the presentinvention.

FIG. 3 shows a simplified block diagram of a portable electronic deviceconstructed and used in accordance with one embodiment of the presentinvention. Portable electronic device 300 may be implemented in or asany type of portable electronic device or devices, such as, for example,portable electronic devices 102 and 202 discussed above.

In some embodiments, portable electronic device 300 can be a portablecomputing device dedicated to processing multi-media data files andpresenting that processed data to the user. For example, device 300 canbe an iPod available by Apple Inc., of Cupertino, Calif., a dedicatedmedia player (e.g., MP3 player), a game player, a remote controller, aportable communication device, a remote ordering interface, or othersuitable personal device. In some embodiments, portable electronicdevice 300 can be a portable device dedicated to providing multi-mediaprocessing and telephone functionality in single integrated unit, suchas an iPhone available from Apple Inc.

Portable electronic device 300 may be battery-operated and highlyportable so as to allow a user to listen to music, play games or videos,record video or take pictures, place and take telephone calls,communicate with other people or devices, control other devices, and anycombination thereof. In addition, portable electronic device 300 may besized such that it fits relatively easily into a pocket or hand of theuser. By being handheld, portable electronic device 300 is relativelysmall and easily handled and utilized by its user and thus may be takenpractically anywhere the user travels.

Portable electronic device 300 can include processor 302, storage 304,user interface 306, display 308, memory 310, input/output circuitry 312,communications circuitry 314, identification module 316, and/or bus 318.In some embodiments, portable electronic device 300 can include morethan one of each component or circuitry, shown in FIG. 3, but for thesake of clarity and illustration, only one of each is shown in FIG. 3.In addition, persons skilled in the art will appreciate that thefunctionality of certain components and circuitry may be combined oromitted and that additional components and circuitry, which are notshown in FIG. 3, may be included in portable electronic device 300.

Processor 302 can include, for example, circuitry for and be configuredto perform any function. Processor 302 may be used to run operatingsystem applications, media playback applications, media editingapplications, and/or any other application. Processor 302 can drivedisplay 308 and can receive user inputs from user interface 306.

Storage 304 can be, for example, one or more storage mediums, includingfor example, a hard-drive, flash memory, permanent memory such as ROM,semi-permanent memory such as RAM, any other suitable type of storagecomponent, or any combination thereof. Storage 304 can store, forexample, media data (e.g., music and video files), application data(e.g., for implementing functions on device 200), firmware, preferenceinformation data (e.g., media playback preferences), lifestyleinformation data (e.g., food preferences), exercise information data(e.g., information obtained by exercise monitoring equipment),transaction information data (e.g., information such as credit cardinformation), wireless connection information data (e.g., informationthat may enable device 200 to establish a wireless connection),subscription information data (e.g., information that keeps track ofpodcasts or television shows or other media a user subscribes to),contact information data (e.g., telephone numbers and email addresses),calendar information data, any other suitable data, or any combinationthereof.

User interface 306 can allow a user to interact with portable electronicdevice 300. For example, the device for user interface 306 may take avariety of forms, such as at least one a button, keypad, dial, a clickwheel, a touch screen or any combination thereof.

Display 308 can accept and/or generate signals for presenting mediainformation (textual and/or graphic) on a display screen, such as thosediscussed above. For example, display 308 may include a coder/decoder(CODEC) to convert digital media data into analog signals. Display 308also can include display driver circuitry and/or circuitry for drivingdisplay driver(s). The display signals can be generated by processor 302or display 308. The display signals can provide media informationrelated to media data received from communications circuitry 314 and/orany other component of portable electronic device 300. In someembodiments, display 308, like any other component discussed herein, maybe integrated with and/or externally coupled to portable electronicdevice 300.

Memory 310 can include one or more different types of memory which maybe used for performing device functions. For example, memory 310 caninclude cache, Flash, ROM, RAM, or one or more different types of memoryused for temporarily storing data. Memory 310 may be specificallydedicated to storing firmware. For example, memory 310 can be providedfor storing firmware for device applications (e.g., operating system,user interface functions, and processor functions).

Input/output circuitry 312 can convert (and encode/decode, if necessary)data, analog signals and other signals (e.g., physical contact inputs,physical movements, analog audio signals, etc.) into digital data, andvice-versa. The digital data may be provided to and received fromprocessor 302, storage 304, memory 310, or any other component ofportable electronic device 300. Although input/output circuitry 312 isillustrated in FIG. 3 as a single component of portable electronicdevice 300, a plurality of input/output circuitry may be included inportable electronic device 300. Input/output circuitry 312 can be usedto interface with any input or output component, such as those discussedin connection with FIGS. 1 and 2. For example, portable electronicdevice 300 can include specialized input circuitry associated with inputdevices such as, for example, one or more microphones, cameras,proximity sensors, accelerometers, ambient light detectors, etc.Portable electronic device 300 can also include specialized outputcircuitry associated with output devices such as, for example, one ormore speakers, etc.

Communications circuitry 314 can permit portable electronic device 300to communicate with one or more servers or other devices using anysuitable communications protocol. For example, communications circuitry314 may support Wi-Fi (e.g., a 802.11 protocol), Ethernet, Bluetooth™(which is a trademark owned by Bluetooth Sig, Inc.), high frequencysystems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems),infrared, TCP/IP (e.g., any of the protocols used in each of the TCP/IPlayers), HTTP, BitTorrent, FTP, RTP, RTSP, SSH, any other communicationsprotocol, or any combination thereof.

Identification module 316 can include any suitable type of technologyfor detecting and identifying objects. For example, identificationmodule 316 may include radio-frequency identification (RFID) tagreaders, infrared (IR) image capture device, a camera capable ofidentifying objects, barcode scanner, biometric reader, proximity cardreader, electronic product code reader, any other detection technology,or any combination thereof. In some embodiments, identification module316 can activate a camera that is operative for capturing images ofobjects to be identified. In some embodiments, identification module 316can activate an IR image capture device (e.g., camera or sensors) thatis operative for capturing IR images of the objects to be identified.Persons skilled in the art will appreciate that identification module316 can use any suitable pattern recognition algorithms to identifyobjects.

In some embodiments, identification module 316 can activate a RFID tagreader that is operative for detecting RFID tags that are located onobjects. Identification module 316 may be operative to read passive,active, and/or semi-passive RFID tags. For example, while the user islooking at pieces of art in an art museum, identification module 316 canactivate the RFID tag reader to read passive RFID tags. In response tothe activation, the RFID tag reader can generate a query to passive RFIDtags that are attached to pieces of art nearby. The RFID tags canrespond to the query by generating radio frequency signals back to theRFID reader.

Bus 318 may provide a data transfer path for transferring data to, from,or between processor 302, storage 304, user interface 306, display 308,memory 310, input/output circuitry 312, communications circuitry 314,and identification module 316.

FIG. 4 shows a schematic view of a communications system which can beoperated in accordance with one embodiment of the invention.Communications system 400 can include portable electronic device 402,and communications network 410, which portable electronic device 402 mayuse to perform wireless communications with other remote servers withincommunications network 410 (e.g., remote server 404). Althoughcommunications system 400 can include several of communications devices402, remote servers 404, and hosts 420, only one of each is shown inFIG. 4 to avoid overcomplicating the drawing.

Any suitable circuitry, device, system or combination of these (e.g., awireless communications infrastructure including communications towersand telecommunications servers) operative to create a communicationsnetwork can be used to create communications network 410. Communicationsnetwork 410 may be capable of providing wireless communications usingany suitable short-range or long-range communications protocol. In someembodiments, communications network 410 can support, for example, Wi-Fi(e.g., an 802.11 protocol), Bluetooth (registered trademark), highfrequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communicationsystems), infrared, other relatively localized wireless communicationprotocol, or any combination thereof.

In some embodiments, communications network 410 can support protocolsused by wireless and cellular phones and personal email devices (e.g., aBlackberry®). Such protocols can include, for example, GSM, GSM plusEDGE, CDMA, quadband, and other cellular protocols. In another example,a long range communications protocol can include Wi-Fi and protocols forplacing or receiving calls using VOIP or LAN. Portable electronic device402 and remote server 404, when located within communications network410, can wirelessly communicate over a local wireless communication pathsuch as path 440.

Portable electronic device 402 and remote server 404 can be any suitabledevice for sending and receiving communications. For example, portableelectronic device 402 and remote server 404 can include a media playersuch as an iPod available by Apple Inc., of Cupertino, Calif., acellular telephone, a personal e-mail or messaging device (e.g., aBlackberry® or a Sidekick®), an iPhone available from Apple Inc.,pocket-sized personal computers such as an iPAQ Pocket PC available byHewlett Packard Inc., of Palo Alto, Calif., personal digital assistants(PDAs), a desktop computer, a laptop computer, and any other devicecapable of communicating wirelessly (with or without the aid of awireless enabling accessory system). The communications sent andreceived can be any suitable form of communications, including forexample, voice communications (e.g., telephonic communications), datacommunications (e.g., e-mails, text messages, media messages), orcombinations of these.

In some embodiments, portable electronic device 402 or remote server 404can be coupled to host device 420 for data transfers, synching theportable electronic device, software or firmware updates, or performingany other suitable operation that may require portable electronic device402 and host device 420 to be coupled. In some embodiments, severalportable electronic devices 402 can be coupled to host 420 to share datausing host 420 as a server. In some embodiments, portable electronicdevice 402 can be coupled to several host devices 420 (e.g., for each ofthe plurality of host devices 420 to serve as a backup for data storedin portable electronic device 402).

Portable electronic device 402 can be coupled with host device 420 overcommunications link 442 using any suitable approach. For example,portable electronic device 402 can use any suitable wirelesscommunications protocol to connect to host device 420 overcommunications link 442. As another example, communications link 442 maybe a wired link that is coupled to both portable electronic device 402and host device 420. As still another example, communications link 442can include a combination of wired and wireless links (e.g., anaccessory device for wirelessly communicating with portable electronicdevice 402 is coupled by a wire to host device 420). Any suitableconnector, dongle or docking station may be used to couple portableelectronic device 402 and host device 420. A more detailed discussion ofa dongle or docking station that can be plugged into portable electronicdevice 402 may be found, for example, in Fadell et al., U.S. patentapplication publication No. 2004-0224638, filed Apr. 25, 2003, entitled“Media Player System,” which is incorporated by reference in itsentirety.

FIG. 5 shows a schematic view of an illustrative display screen of ahome screen in accordance with one embodiment of the invention. Displayscreen 500 may include icons 502 representing applications that the usermay access. The applications may include text, calendar, photos, camera,phone, mail, or any other suitable application that a portableelectronic device, such as portable electronic device 300 (FIG. 3), mayexecute. Display screen 500 may be displayed on the portable electronicdevice when the portable electronic device is first turned on or whenthe portable electronic device has exited out of an application that wasrunning. Display screen 500 may also be displayed when a user pressesbutton 504 while the portable electronic device is still running anapplication.

The user may select an icon 502 by placing highlight region 506 over anicon. In some embodiments, in response to receiving a user selection ofan identification application, the portable electronic device mayactivate an identification module, such as identification module 316(FIG. 3) (in fact, highlight region 506 in FIG. 5 is shown to behighlighting an Identification Application).

FIG. 6 shows a schematic view of an illustrative display screen forselecting objects to identify in accordance with one embodiment of theinvention. Display screen 600 can include identification screen 602. Insome embodiments, identification screen 602 can display images as seenthrough the camera lens. For example, the user may use identificationscreen 602 to locate one or more objects to be identified. In someembodiments, identification screen 602 can display IR images as seenthrough the IR camera on the portable electronic device. In someembodiments, identification screen 602 can include messages for usingthe portable electronic device to detect objects that are labeled withRFID tags. An example of a message can be “Select GO to identifyobjects.”

In some embodiments, display screen 600 can include “SETTINGS” option604. In response to the user selecting “SETTINGS” option 604, theportable electronic device can provide configuration options to theuser.

FIG. 7 shows a schematic view of an illustrative display screen forproviding configuration options in accordance with one embodiment of theinvention. Display screen 700 can include listings 702, which can allowthe user to specify a detection technology that will be used to detectobjects. In some embodiments, the user may manually select to use adetection technology (e.g., “RFID,” “CAMERA,” or “IR CAMERA”). Forexample, in response to the user selecting “RFID” by placing highlightregion 704 over listings 702, the portable electronic device can use aRFID reader to detect objects in the museum.

In some embodiments, in response to the user selecting “AUTO” inlistings 702, the portable electronic device can automatically detectthe current environment and select a detection technology that iswell-suited for the detected environment (or the system may beconfigured to user the “AUTO” setting as a default). For example, inresponse to detecting that the portable electronic device is being usedin the dark, the portable electronic device can activate an IR imagecapture device that is capable of detecting objects in darkness. Asanother example, if the portable electronic device detects active RFIDtags broadcasting signals to the RFID reader, the portable electronicdevice can use the RFID reader to detect objects.

In some embodiments, display screen 700 can include listings 706, whichcan allow the user to select a mode based on the types of objects thatthe user wants to identify. In some embodiments, modes can beimplemented as metadata tags that are associated with images of objectsor RFID tags of objects. For example, an image of an object can betagged with both “GLOBAL” and “ZOO” modes. Depending on the selectedmode, the portable electronic device can adjust parameters that are usedto search an identification database. The identification database may belocated in local storage (e.g., storage 304), an external source (e.g.,remote server 404), or any combination thereof. The identificationdatabase can include one or more image databases, IR image databases,photo libraries, RFID identification databases, or any other suitabletype of identification databases.

In some embodiments, the identification database can include images oflabeled objects (e.g., people, animals, buildings, and other objects).For example, in response to the user selecting “MUSEUM” mode by placinghighlight region 708 over listings 706, the portable electronic devicecan limit the search to images of objects that are commonly found in amuseum (e.g., paintings, sculptures, and illustrations). As anotherexample, in response to the user selecting “PEOPLE” mode, the portableelectronic device can limit the search to images of people. As stillanother example, in response to the user selecting “GLOBAL” mode inlistings 706, the portable electronic device can perform a search overthe entire identification database in order to identify an object.

The portable electronic device can use one or more pattern recognitionalgorithms to compare the detected object with images in theidentification database. For example, suitable types of patternrecognition algorithms can include neural networks, support vectormachines, decision trees, K-nearest neighbor, Bayesian networks, MonteCarlo methods, bootstrapping methods, boosting methods, or anycombination thereof.

In some embodiments, if the user has selected to use “RFID” detectiontechnology, the portable electronic device can access a RFIDidentification database. The RFID identification database can includeobject names corresponding to particular RFID tags. For example, RFIDtag “ID_04584” may correspond to the painting “On the Terrace.”

In some embodiments, display screen 700 can include “EDIT” option 710,which allows the user to edit modes listed in listings 706. In responseto the user selecting “EDIT” option 710, the user may add new modes ordelete existing modes. In some embodiments, in response to the userselecting “EDIT” option 710, the portable electronic device can producelists of objects that are currently associated with the various modes.The user may have the option to change these associations. For example,after identifying an object in “MUSEUM” mode, the user may decide thatthe object should also be associated with “FLOWER” mode. The user mayselect “EDIT” option 710 in order to associate the object with both“FLOWER” and “MUSEUM” modes.

Referring back to FIG. 6, in response to a user selecting “GO” option606, the portable electronic device can attempt to identify objects thatare able to be detected. In some embodiments, in addition to using theselected mode to assist in object identification, the portableelectronic device can use a GPS receiver, RF triangulation detector orsensor, or any other location circuitry configured, to locate the user.For example, the user may have taken a picture of a restaurant called“Lou's Diner” while in the “RESTAURANT” mode. In response to locatingthat the user is in Las Vegas, Nev., the portable electronic device canlimit the search of the identification database to “Lou's Diner”restaurants in Las Vegas.

FIG. 8 shows a schematic view of an illustrative display screen of alisting of objects that have been detected and/or identified inaccordance with one embodiment of the invention. Display screen 800 mayinclude listings 802 of objects (e.g., paintings, artifacts, buildings)that have been detected and/or identified by the portable electronicdevice. For example, while the user is looking at pieces of art in anart museum, the user may select to activate the RFID tag reader. TheRFID tag reader can detect RFID tags that are attached to paintingsnearby and display the RFID tags in listings 802. The RFID tag readermay also access a RFID identification database and look for thecorresponding object names.

In some embodiments, the portable electronic device can provide “EDIT”option 804 to the user. In some embodiments, in response to receiving auser selection of “EDIT” option 804, the portable electronic device canallow the user to add or change the identity of an object.

FIG. 9 is a schematic view of an illustrative display screen for editingan object in accordance with one embodiment of the invention. In someembodiments, display screen 900 can include media screen 902, which candisplay an image of the object. In some embodiments, if the image ismissing or unsatisfactory, the user may take a picture of the object byselecting media screen 902. In some embodiments, the user may identifythe object or change the identity of the object by selecting highlightregion 904. For example, if the portable electronic device was unable toidentify the object or if the portable electronic device incorrectlyidentified the object, the user may select highlight region 904 tochange the label for the object.

In some embodiments, if the portable electronic device finds multiplecandidate identifications for an object, the portable electronic devicemay list all of the multiple candidate identifications in display screen900. The user may then choose the identity of the object among thecandidates identifications or input a new identification.

In some embodiments, if the portable electronic device finds multiplecandidate identifications for an object, the portable electronic devicemay list the candidate identification that the device believes is thebest identification match for the object based on certain criteria. Theuser can be given the ability to directly modify the criteria, and/orthe criteria may be self-modifying based on user input. For example, ifthe user agrees with the best identification match, the user may selectthe best identification match as the identity of the object. However, ifthe user disagrees with the best match, the user may input a correctedidentity for the object by selecting highlight region 904. In responseto the correction, the portable electronic device can use the user inputas training data for future identifications.

In some embodiments, the criteria can include support vector machines,in which each support vector machine can be trained to identify acertain object. In response to the user input, the support vectormachines can be re-trained based on an enlarged data set, which includesthe image of the object combined with the corrected identity. Personsskilled in the art will appreciate that other self-modifying criteriacan be used without departing from the spirit of the present invention.

In some embodiments, the user may select “ADD” option 906 to add theidentified object to the identification database. In some embodiments,the user may select “ADD” option 906 to change the mode association(s)of the identified object.

Returning to display screen 800, the user may select a listing byplacing highlight region 806 over listings 802. In some embodiments, inresponse to receiving a user selection of an object, the portableelectronic device can search and display information related to theobject. In some embodiments, the information acquired can haveincrementally increasing levels of detail (e.g., basic, detailed, andextended information).

The information can be located locally, on the device itself (e.g.,storage 304), on a local server, on an external source (e.g., a remoteserver connected to the internet), or any combination of these. In someembodiments, the information can be stored on an information database onthe device, the local server, and/or the external source. In someembodiments, the information search can be performed based on thepreviously-defined mode (e.g., the selected mode in listings 706). Forexample, if the user is in “MUSEUM” mode, the search can be restrictedto museum related information.

In some embodiments, if the portable electronic device detects that itcan not connect to an appropriate communications line to access remoteinformation (e.g., lacks WI-FI connectivity), the device can accessinformation stored locally. For example, when the user requests forinformation related to an identified object, the portable electronicdevice can search for information that is locally stored on the deviceor on a local server.

In some embodiments, the portable electronic device can first downloadinformation from an external source before providing the relevantinformation to the user. For example, when a user goes to a museum, theuser can plug the portable electronic device in to a suitable connector(e.g., docking station). Once the device is connected, informationrelated to each of the exhibits in the museum can be downloaded andstored locally on the portable electronic device. The user can then usethe portable electronic device to identify pieces of art in the museum.When the portable electronic device identifies that the user is viewinga Monet painting, for example, the device can provide information thathas been downloaded about the Monet exhibit.

FIG. 10 shows a schematic view of an illustrative display screen forproviding basic information related to an identified object inaccordance with one embodiment of the invention. Display screen 1000 mayinclude media screen 1002, which may present an image of the identifiedobject captured using the device's camera. In some embodiments, theimage of the detected object presented in media screen 1002 can beobtained from a local and/or external source.

In some embodiments, display screen may include basic information region1004. For example, in response to a user selecting a painting inlistings 802 (FIG. 8) in “MUSEUM” mode, the portable electronic devicecan display an image of the painting in media screen 1002. In addition,the portable electronic device can provide the name, date, and artist ofthe painting in information region 1004. As another example, in responseto a user selecting a restaurant in listings 802 in “RESTAURANT” mode,the portable electronic device can display an image of the front of therestaurant in media screen 1002. In addition, the portable electronicdevice can provide selectable options to the user, which can includecontact information (e.g., phone number and website) and locationinformation of the restaurant. In some embodiments, in response to theuser selecting a “DIAL” option, the portable electronic device can allowthe user to call the restaurant. In some embodiments, in response to theuser selecting a “LOCATE” option, the portable electronic device canlocate the restaurant on a map application (e.g., Google Maps™).

FIG. 11 shows a schematic view of an illustrative display screenproviding a listing of detailed information related to an identifiedobject in accordance with one embodiment of the invention. Displayscreen 1100 can include listings 1102, which may include informationstored both locally and/or remotely on an external source. Relatedinformation can include all types of media, such as text, images,clipart, video clips, movies, and any other suitable relatedinformation.

The user may select a listing 1102 by placing highlight region 1104 overa listing. In some embodiments, in response to receiving a userselection of a listing, the portable electronic device may access thedetailed information and provide the detailed information to the user.

FIG. 12 shows a schematic view of an illustrative display screen ofdetailed information that includes media in accordance with oneembodiment of the invention. Display screen 1200 can include mediascreen 1202, which can display media related to the object. In someembodiments, media screen 1202 can include a series of images that theuser may flip through by accessing a user interface, such as userinterface 306 (FIG. 3), on the portable electronic device. In someembodiments, media screen 1202 can display a video. The portableelectronic device can provide various media control functions (e.g.,play, stop, pause, or any other suitable media control functions), whichthe user may use to control the media. In some embodiments, media screen1202 can display text that provides information related to the object.For example, media screen 1202 can display word definitions orinformation gathered from an electronic encyclopedia (e.g., Wikipedia®).In some embodiments, a mixture of different types of media can bedisplayed in media screen 1202.

The user may select description region 1204 to find additional detailedinformation about the media displayed in media screen 1202. In someembodiments, in response to receiving a user selection of descriptionregion 1204, the portable electronic device can display additionaldetailed information related to the media displayed in media screen1202.

Returning to display screen 1100, the user may select “EXTENDEDINFORMATION” option 1106 to view extended information about theidentified object. In some embodiments, in response to receiving a userselection of “EXTENDED INFORMATION” option 1106, the portable electronicdevice can display extended information about the object.

FIG. 13 shows a schematic view of an illustrative display screen ofextended information related to an object in accordance with oneembodiment of the invention. Display screen 1300 can include links 1302,which can provide the user with extended information related to theidentified object located on a local source, such as storage 304 (FIG.3), or an external source, such as remote server 404 (FIG. 4). Theportable electronic device can provide links 1302 by conducting a websearch based on the identified object and the previously-defined mode.For example, if the identified object is “lily” and thepreviously-defined mode is “FLOWER,” the portable electronic device cansearch for “lily flower.”

In some embodiments, the user may select a link within links 1302 toaccess a webpage located on a server connected to the Internet. In someembodiments, in response to the user selecting a link, the portableelectronic device can display the webpage on a web browser (such asSafari available from Apple Inc.).

Returning to display screen 600, the user may select to view a log,which may include a history of identified objects. In some embodiments,in response to receiving a user selection of “LOG” option 608, theportable electronic device may display a log of identified objects.

FIG. 14 shows a schematic view of an illustrative display screen forproviding a log of identified objects in accordance with one embodimentof the invention. Display screen 1400 can include listings 1402, whichcan include a history of previously identified objects. Listings 1402can include different types of objects (e.g., paintings, people,animals, and buildings). Each listing in listings 1402 can also includethe date that the object was identified.

The user may select a listing by placing highlight region 1404 overlistings 1402. In some embodiments, in response to receiving a userselection of a listing in listings 1402, the portable electronic devicecan search and display information related to the identified object. Forexample, the user can be taken to a display screen similar to or thesame as display screen 1000 (FIG. 10), where the user may requestdisplay of information with incrementally increasing levels of detail.

In some embodiments, displays screen 1400 can include “CUSTOMIZE” option1406. For example, in response to the user selecting “CUSTOMIZE” option1406, the portable electronic device can delete objects in the log thathave not been accessed for a period of time (e.g. one year). The usercan be provided with the opportunity to set the operationalcharacteristics of the log in a “Preferences” or “System Settings”option instead, which may be accessed from the home age, such as the“Settings” icon shown in FIG. 5.

In some embodiments, the user may select “CUSTOMIZE” option 1406 toarchive certain objects before the objects are deleted. In someembodiments, the user may select “CUSTOMIZE” option 1406 toautomatically archive the objects that are deleted from the log. In someembodiments, instead of displaying the identified objects in the log bydate, the user may select “CUSTOMIZE” option 1406 to sort the identifiedobjects based on previously-defined modes.

In some embodiments, the user may select “CUSTOMIZE” option 1406 tocreate a multi-media album that includes objects that were identified ina previously-defined mode or modes. For example, the user may have usedthe portable electronic device to identify objects in the “MUSEUM” modeon a few museum visits. As a result, the user can create a multi-mediaalbum by collecting all of the identified objects in the “MUSEUM” mode.The user may let his friend borrow the portable electronic device orupload the multi-media album to his friend's portable electronic device.After receiving the multi-media album, the friend may then create aself-guided tour that organizes the pieces of art in a desired order.

By selecting a particular piece of art, the friend may see basicinformation, such as the title of the art, the date, and the artist. Thefriend may find additional information about the art by requesting asearch of detailed and extended information. It will be understood thatalthough the previous discussion involved certain customizable features,other suitable customizable features can also be implemented in thepresent invention.

FIGS. 15A-15D show a flowchart of an illustrative process for providinginformation related to an identified object in accordance with oneembodiment of the invention. Process 1500 begins at step 1502. At step1504, a portable electronic device can begin to receive userinstructions. For example, the portable electronic device can providedisplay screen 600 (FIG. 6) to a user. At step 1506, the portableelectronic device can determine whether an instruction to select a newdetection technology has been received. For example, the portableelectronic device can determine whether the user has selected “SETTINGS”option 604 and a detection technology in listings 702 (FIG. 7).

If, at step 1506, the portable electronic device determines that aninstruction to select a new detection technology has not been received,process 1500 moves to step 1508. If, at step 1506, the portableelectronic device instead determines that an instruction to select a newdetection technology has been received, process 1500 moves to step 1510.At step 1510, the portable electronic device can select the newdetection technology based on user instructions. For example, the usermay manually select a detection technology (e.g., “CAMERA,” “RFID,” or“IR CAMERA”). The user may also select “AUTO” in listings 702, so thatthe portable electronic device can automatically determine the currentenvironment of the portable electronic device. In response todetermining the current environment, the portable electronic device canselect to use a detection technology that is well-suited for theenvironment. The user may save the changes by selecting “BACK” option712. Process 1500 then moves to step 1508.

At step 1508, the portable electronic device can determine whether aninstruction to select a new mode has been received. For example, theportable electronic device can determine whether the user has selected“SETTINGS” option 604 and a new mode in listings 706 (FIG. 7). If, atstep 1508, the portable electronic device determines that an instructionto select a new mode has not been received, process 1500 moves to step1512. If, at step 1508, the portable electronic device insteaddetermines that an instruction to select a new mode has been received,process 1500 moves to step 1514. At step 1514, the portable electronicdevice can change the mode to the selected mode. The user may save thechanges by selecting “BACK” option 712. Process 1500 the moves to step1512.

At step 1512, the portable electronic device can determine whether aninstruction to identify an object has been received. If, at step 1512,the portable electronic device determines that an instruction toidentify an object has been received, process 1500 moves to step 1516.For example, the user may select to identify an object by selecting “GO”option 606 (FIG. 6). At step 1516, the portable electronic device candetect and/or identify the object. In response to the user selecting“GO” option 606, the portable electronic device can detect the objectusing the detection technology. After detecting the object, the portableelectronic device can search an identification database to attempt toidentify the detected object based on the selected mode.

At step 1518, the portable electronic device can display the detectedand/or identified object. For example, the portable electronic devicecan direct the user to a display screen such as display screen 800 (FIG.8), which includes listings 802. The portable electronic device candisplay the detected and/or identified object in listings 802.

At step 1520, the portable electronic device can determine whether theobject was correctly identified. If, at step 1520, the portableelectronic device determines that the object was correctly identified,process 1500 moves to step 1522.

If, at step 1520, the portable electronic device instead determines thatthe object was incorrectly identified, process 1500 moves to step 1524.At step 1524, the portable electronic device can correct the object'sidentity. For example, in response to the user selecting “EDIT” option804 (FIG. 8), the user can be directed to a display screen such asdisplay screen 900, which includes highlight region 904. The user canselect highlight region 904 to change the object's identity. Aftercorrecting the object's identity, process 1500 moves to step 1522. Atstep 1522, the portable electronic device can store the identifiedobject in the identification database. For example, the user can storethe identified object in the database by selecting “ADD” option 906.

After the portable electronic device finishes storing the identifiedobject, process 1500 then moves to step 1526. At step 1526, the portableelectronic device can search for basic information related to theidentified object. For example, in response to the user selecting anidentified object in listings 802 (FIG. 8), the portable electronicdevice can search for basic information that can provide pertinentinformation about the object. At step 1528, the portable electronicdevice can provide the basic information to the user. For example, theuser can be directed to a display screen such as display screen 1000(FIG. 10), which can include media screen 1002. In some embodiments,display screen 1000 can include basic information region 1004.

At step 1530, the portable electronic device can determine whether ithas received an instruction to search for detailed information. Forexample, the user may select basic information region 1004 to requestdetailed information, which can provide additional information relatedto the object. If, at step 1530, the portable electronic devicedetermines that it has received instruction to search for detailedinformation, process 1500 moves to step 1532. At step 1532, the portableelectronic device can search for detailed information related to theidentified object. After the portable electronic device has searched fordetailed information, process 1500 then moves back to step 1528. At step1528, the portable electronic device can provide the detailedinformation to the user. For example, the portable electronic device maydirect the user to a display screen such as display screen 1100 (FIG.11), which includes listings 1104. The user may select a listing inlistings 1104 to access detailed information related to the identifiedobject. An example of detailed information is provided in display screen1200 (FIG. 12).

If, at step 1530, the portable electronic device instead determines thatit has not received an instruction to search for detailed information,process 1500 then moves to step 1534. At step 1534, the portableelectronic device can determine whether it has received an instructionto search for extended information. For example, the user may select“EXTENDED INFORMATION” option 1106 (FIG. 11) to request extendedinformation. Extended information can include links that are consideredrelevant to the object.

If, at step 1534, the portable electronic device determines that it hasnot received an instruction to search for extended information, process1500 moves back to step 1504 (FIG. 15A), where the portable electronicdevice can continue to receive user instructions. For example, the usermay have selected “BACK” options in FIGS. 8-13 in order to return todisplay screen 600 (FIG. 6). If, at step 1534, the portable electronicdevice instead determines that it has received an instruction to searchfor extended information, process 1500 moves to step 1536. At step 1536,the portable electronic device can search for extended informationrelated to the identified object. After the portable electronic devicehas searched for the extended information, process 1500 moves back tostep 1528. At step 1528, the portable electronic device can provide theextended information to the user. For example, the portable electronicdevice may direct the user to a display screen such as display screen1300 (FIG. 13), which includes listings 1302. The user may select alisting in listings 1302 to access extended information related to theidentified object.

Referring back to FIG. 15A, if, at step 1512, the portable electronicdevice instead determines that an instruction to identify an object hasnot been received, process 1500 moves to step 1538. At step 1538, theportable electronic device may determine whether it has received aninstruction to provide a log. For example, the user may select “LOG”option 608 (FIG. 6) to request a history of identified objects. If, atstep 1538, the portable electronic device determines that an instructionto provide a log has not been received, process 1500 ends at step 1540.

If, at step 1538, the portable electronic device instead determines thatan instruction to provide a log has been received, process 1500 moves tostep 1542. At step 1542, the portable electronic device may provide alog of the identified objects to the user. For example, the portableelectronic device may direct the user to a display screen such asdisplay screen 1400 (FIG. 14), which includes listings 1402 ofidentified objects.

At step 1544, the portable electronic device can determine if it hasreceived a selection of an identified object in the log. For example,the user may select an identified object in listings 1402 by placinghighlight region 1404 over a listing. If the portable electronic devicedetermines that it has received a selection of an identified object,process 1500 returns to step 1526, as discussed previously. If theportable electronic device determines that it has not received aselection of an identified object, process 1500 moves to step 1546. Atstep 1546, the portable electronic device can determine if it hasreceived a selection of a customized option. For example, the user mayselect “CUSTOMIZE” option 1406 to customize the log.

If, at step 1546, the portable electronic device determines that it hasnot received a selection of a customized option, process 1500 moves backto step 1504 (FIG. 15A), as discussed previously. For example, the usermay select “BACK” option 1408 (FIG. 14) to return to display screen 600(FIG. 6). If, at step 1546, the portable electronic device insteaddetermines that it has received a selection of a customized option,process 1500 moves to step 1548. At step 1548, the portable electronicdevice can determine whether it has received an instruction to archivecertain identified objects.

If, at step 1548, the portable electronic device determines that it hasnot received an instruction to archive certain identified objects,process 1500 moves to step 1550. If, at step 1548, the portableelectronic device instead determines that it has received an instructionto archive certain identified objects, process 1500 moves to step 1552.At step 1552, in response to receiving the instruction, the portableelectronic device can allow the user to archive those objects. Afterarchiving the objects, process 1500 moves to step 1550.

At step 1550, the portable electronic device can determine whether ithas received an instruction to delete identified objects if the objectshave not been accessed for a period of time. If, at step 1550, theportable electronic device determines that it has not received aninstruction to delete identified objects, process 1500 moves to step1554. If, at step 1550, the portable electronic device insteaddetermines that it has received an instruction to delete identifiedobjects, process 1500 moves to step 1556. At step 1556, in response toreceiving the instruction, the portable electronic device can monitorthe deletion of identified objects if the objects have not been accessedfor a period of time. After the portable electronic device has beenconfigured to monitor the deletion of these objects, process 1500 movesto step 1554.

At step 1554, the portable electronic device can determine whether ithas received an instruction to sort the identified objects based onpreviously-defined modes. If, at step 1554, the portable electronicdevice determines that it has not received an instruction to sort theidentified objects, process 1500 moves to step 1558. If, at step 1554,the portable electronic device instead determines that it has receivedan instruction to sort the identified objects, process 1500 moves tostep 1560. At step 1560, in response to receiving the instruction, theportable electronic device can allow the user to sort the identifiedobjects in the log based on the previously-defined modes. After sortingthe identified objects, process 1500 moves to step 1558.

At step 1558, the portable electronic device can determine whether ithas received an instruction to create a multi-media album that includesobjects that were identified in a previously-defined mode or modes. If,at step 1558, the portable electronic device determines that it has notreceived an instruction to create a multi-media album, process 1500moves back to step 1542 to provide the log to the user (FIG. 15C). If,at step 1558, the portable electronic device instead determines that ithas received an instruction to create a multi-media album, process 1500moves to step 1562. At step 1562, the-portable electronic device canallow the user to create a multi-media album. After the user has createdthe multi-media album, process 1500 moves back to step 1542.

Thus it is seen that systems and methods are provided for identifyingobjects and obtaining additional information about the identifiedobjects. It is also seen that systems and methods are provided forviewing a history of previously identified objects. Persons skilled inthe art will appreciate that the invention can be practiced by otherthan the described embodiments, which are presented for purposes ofillustration and not of limitation, and the present invention is limitedonly by the claims which follow.

1.-25. (canceled)
 26. A system comprising: a data source; and anelectronic device comprising: an output component; object detectiontechnology operative to detect an object in an environment of theelectronic device; location detection technology operative to detect thelocation of the electronic device when the object is detected; anidentification module operative to communicate with the data source forobtaining identification information from the data source about thedetected object; and a processor operative to present the identificationinformation for the detected object to a user via the output component,wherein the detected location of the electronic device when the objectis detected specifies at least one parameter for searching the datasource for obtaining the identification information from the datasource.
 27. The system of claim 26, wherein the object is detected basedon a previously-defined mode selected by the user, and wherein thepreviously-defined mode specifies at least one other parameter forsearching the data source for obtaining the identification informationfrom the data source.
 28. The system of claim 26, wherein the datasource is remote from the electronic device.
 29. The system of claim 26,wherein the identification information comprises at least oneuser-selectable option for providing additional information about thedetected object.
 30. The system of claim 26, wherein the identificationinformation comprises at least two user-selectable options for providingdifferent types of additional information about the detected object. 31.The system of claim 26, wherein the object detection technologycomprises a camera, and wherein the output component comprises a visualdisplay.
 32. The system of claim 26, wherein the location detectiontechnology comprises a global positioning system receiver.
 33. Thesystem of claim 26, wherein the data source comprises an informationdatabase of multi-media data from which the identification informationis obtained using a search based on the detected location of theelectronic device when the object is detected.
 34. The system of claim26, wherein the processor is operative to simultaneously present theidentification information for the detected object and an image of thedetected object via the output component.
 35. The system of claim 34,wherein the identification information comprises at least twouser-selectable options.
 36. The system of claim 35, wherein each one ofthe at least two user-selectable options is operative to provideadditional unique information associated with the detected object whenselected by the user.
 37. The system of claim 34, wherein theidentification information comprises at least one link to a websitecomprising additional information associated with the detected object.38. The system of claim 34, wherein the object detection technologycomprises a camera operative to capture the image, and wherein theoutput component comprises a visual display.
 39. The system of claim 26,wherein the identification information comprises at least one link to awebsite comprising additional information associated with the detectedobject.
 40. A system comprising: a data source; and an electronic devicecomprising: an output component; object detection technology operativeto detect an object in an environment of the electronic device; locationdetection technology operative to detect a location of the electronicdevice; an identification module operative to communicate with the datasource for obtaining identification information from the data sourceabout the detected object; and a processor operative to present theidentification information for the detected object to a user via theoutput component, wherein the data source comprises an informationdatabase of multi-media data from which the identification informationis obtained using the detected location of the electronic device whenthe object was detected.
 41. A method for operating an electronic devicethat comprises object detection technology, location detectiontechnology, an output component, and a processor, the method comprising:detecting an object in an environment of the electronic device using theobject detection technology; determining a location of the electronicdevice using the location detection technology when the objected wasdetected; in response to the detecting and the determining,automatically obtaining, using the processor, identification informationassociated with both the detected object and the determined location;and presenting data, using the output component, wherein the presenteddata comprises at least a portion of the obtained identificationinformation.
 42. The method of claim 41, further comprising, prior tothe detecting, implementing a mode selected by a user of the electronicdevice, wherein the detecting the object comprises detecting the objectwhile the mode is implemented, and wherein the automatically obtainingthe identification information comprises automatically obtaining theidentification information using the processor and at least oneparameter specified by the implemented mode.
 43. The method of claim 41,wherein the data further comprises a user-selectable option.
 44. Themethod of claim 43, wherein, when the user-selectable option isselected, presenting, using the output component, other data comprisingadditional identification information associated with the detectedobject.
 45. The method of claim 44, wherein the additionalidentification information comprises another portion of the obtainedidentification information.
 46. The method of claim 43, wherein theuser-selectable option comprises a link to a website comprisingadditional information associated with the detected object.
 47. Themethod of claim 41, wherein the automatically obtaining theidentification information comprises automatically obtaining theidentification information from a data source remote from the electronicdevice.
 48. The method of claim 41, wherein the data further comprisesan image of the detected object.